Skip to content

Conversation

@azwanzuharimi
Copy link
Contributor

@azwanzuharimi azwanzuharimi commented Jan 29, 2026

Adding a new data source (AWS Athena).

Login to AWS by:

  • setting AWS profile (as per picture) set during SSO login, or having it in ~/.aws/credentials.
  • key in key/secret
image image

@azwanzuharimi
Copy link
Contributor Author

azwanzuharimi commented Jan 29, 2026 via email

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new external data source integration for AWS Athena, enabling users to run Athena SQL and load query results into DuckDB via S3.

Changes:

  • Introduces AthenaDataLoader with AWS auth (profile/keys), workgroup output discovery, query execution/polling, and result ingestion from S3 into DuckDB.
  • Registers the new loader in the data loader registry (DATA_LOADERS) and exports it via __all__.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
py-src/data_formulator/data_loader/athena_data_loader.py New Athena loader implementation: credentials setup, query execution, S3 result loading into DuckDB, and table listing.
py-src/data_formulator/data_loader/__init__.py Adds AthenaDataLoader import, registry entry, and export list update.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Collaborator

@Chenglong-MS Chenglong-MS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, we'll merge and test backwardly if any issues are spotted at runtime

@Chenglong-MS Chenglong-MS requested a review from Mestway January 29, 2026 19:44
@Chenglong-MS Chenglong-MS changed the base branch from main to dev January 30, 2026 00:39
@Chenglong-MS Chenglong-MS merged commit a7bf6da into microsoft:dev Jan 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants